Remote presence system including a cart that supports a robot face and an overhead camera

ABSTRACT

A tele-presence system that includes a cart. The cart includes a robot face that has a robot monitor, a robot camera, a robot speaker, a robot microphone, and an overhead camera. The system also includes a remote station that is coupled to the robot face and the overhead camera. The remote station includes a station monitor, a station camera, a station speaker and a station microphone. The remote station can display video images captured by the robot camera and/or overhead camera. By way of example, the cart can be used in an operating room, wherein the overhead camera can be placed in a sterile field and the robot face can be used in a non-sterile field. The user at the remote station can conduct a teleconference through the robot face and also obtain a view of a medical procedure through the overhead camera.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The subject matter disclosed generally relates to the field of robotictele-presence systems.

2. Background Information

Robots have been used in a variety of applications ranging from remotecontrol of hazardous material to assisting in the performance ofsurgery. For example, U.S. Pat. No. 5,762,458 issued to Wang et al.discloses a system that allows a surgeon to perform minimally invasivemedical procedures through the use of robotically controlledinstruments. One of the robotic arms in the Wang system moves anendoscope that has a camera. The camera allows a surgeon to view asurgical area of a patient.

There has been marketed a mobile tele-presence robot introduced byInTouch Technologies, Inc., the assignee of this application, under thetrademark RP-7. The InTouch robot is controlled by a user at a remotestation. The remote station may be a personal computer with a joystickthat allows the user to remotely control the movement of the robot. Boththe robot and remote station have cameras, monitors, speakers andmicrophones to allow for two-way video/audio communication. The robotcamera provides video images to a screen at the remote station so thatthe user can view the robot's surroundings and move the robotaccordingly.

InTouch also provides a system sold as VisitOR that includes a robotface that is attached to a boom. The boom and robot face can beinstalled into an operating room. Using a robot face in an operatingroom may require sterilization of the face. Additionally, the VisitORrequires the installation of a boom in the operating room. This can addto the cost and complexity of installing such a system.

BRIEF SUMMARY OF THE INVENTION

A tele-presence system that includes a cart. The cart includes a robotface that has a robot monitor, a robot camera, a robot speaker, a robotmicrophone, and an overhead camera. The system also includes a remotestation that is coupled to the robot face and the overhead camera. Theremote station includes a station monitor, a station camera, a stationspeaker and a station microphone.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a tele-presence system;

FIG. 2 is a perspective view of a cart of the system;

FIG. 3 is a rear view of an articulated arm and a robot face of thecart;

FIG. 4 is an enlarged perspective view of an overhead camera of thecart;

FIG. 5 is an illustration of a display user interface of a remotestation;

FIG. 6 is an illustration of the display user interface showing videoimages captured by a robot camera and an overhead camera beingsimultaneously displayed.

DETAILED DESCRIPTION

Disclosed is a tele-presence system that includes a cart. The cartincludes a robot face that has a robot monitor, a robot camera, a robotspeaker, a robot microphone, and an overhead camera. The system alsoincludes a remote station that is coupled to the robot face and theoverhead camera. The remote station includes a station monitor, astation camera, a station speaker and a station microphone. The remotestation can display video images captured by the robot camera and/oroverhead camera. By way of example, the cart can be used in an operatingroom, wherein the overhead camera can be placed above a sterile field toprovide a more advantageous vantage point to view a procedure. The userat the remote station can conduct a teleconference through the robotface and also obtain a view of a medical procedure through the overheadcamera.

Referring to the drawings more particularly by reference numbers, FIGS.1, 2 and 3 show a tele-presence system 10. The system 10 includes a cart12 that is coupled to a remote control station 14. The cart 12 has arobot face 16 and an overhead camera 18. The remote control station 14may be coupled to the cart 12 through a network 20. By way of example,the network 20 may be either a packet switched network such as theInternet, or a circuit switched network such as a Public SwitchedTelephone Network (PSTN) or other broadband system. Alternatively, thecart 12 may be coupled to the remote station 14 network thru asatellite.

The remote control station 14 may include a computer 22 that has amonitor 24, a camera 26, a microphone 28 and a speaker 30. The computer22 may also contain an input device 32 such as a joystick or a mouse.The control station 14 is typically located in a place that is remotefrom the cart 12. Although only one remote control station 14 is shown,the system 10 may include a plurality of remote stations 14. In generalany number of carts 12 may be coupled to any number of remote stations14 or other carts 12. For example, one remote station 14 may be coupledto a plurality of carts 12, or one cart 12 may be coupled to a pluralityof remote stations 14, or a plurality of carts 12. The system mayinclude an arbitrator (not shown) that control access between the carts12 and the remote stations 14.

As shown in FIG. 3, the cart 12 may include an articulated arm 40 thatsupports and can move the robot face 16. The articulated arm 40 may haveactive joints 42 and 44 that allow the robot face 14 to be panned andtilted, respectively. The active joints 42 and 44 may move in responseto commands provided by the remote station. The joints 42 and 44 maycontain position sensors 46 and 48, respectively, that providepositional feedback of the arm 40.

Referring to FIGS. 2 and 3, each robot face 16 includes a camera(s) 50,a monitor 52, a microphone(s) 54 and a speaker(s) 56. The robot camera50 is coupled to the remote monitor 24 so that a user at the remotestation 14 can view a video image captured by the robot camera 50.Likewise, the robot monitor 52 is coupled to the remote camera 26 sopersonnel at the surgical site may view the user of the remote station14. The microphones 28 and 54, and speakers 30 and 56, allow for audiblecommunication between the system operator and the personnel at thesurgical site.

The overhead camera 18 may be coupled to a boom 60. The boom 60 mayinclude a number of joints 62, either active or passive. The joints 62may include positional sensors to provide feedback regarding theposition of the overhead camera 18.

As shown in FIG. 4, the cart 12 may include an overhead microphone 64and a detachable handle 66. The overhead microphone 64 may provide analternative source of sound. The detachable handle 66 can be used tomove the boom 60 and overhead camera 18. If the cart 12 is used in asterile field, for example in an operating room, the handle 66 may bereplaced with a sterile handle before each medical procedure to allow asurgeon within the sterile field to position the boom during aprocedure.

Referring again to FIG. 2, the cart 12 may include a linear actuator 70that can be remotely or locally actuated to vary the height of the robotface 16 and overhead camera 18. Varying the height allows the cart 12 tobe rolled through doors and then actuated to move the face 16 and camera18 to elevated positions. For example, the face 16 and camera 18 can belowered to allow the cart 12 to be moved into an operating room. Thecamera 18 can then be raised to provide a desirable view over anoperating table. The cart 12 may include a laser pointer 72 and/ordirected lighting (not shown) located on the boom 60. The cart 12 mayalso include a local control panel 74 to move the articulated arm 40,actuator 70 and/or boom 60. The linear actuator 70 is also advantageousin moving the face 16 to be essentially at the same level as a personwhether they are standing, sitting or lying in a prone position.

The robot face 16 may include a processor, hard disk drive and othercircuits that enable the face 16 to function as a computer. The face 16may include an input panel 76 that allows a user to provide input. Byway of example, the operator of the remote station may provide one ormore questions through the robot face 16, wherein a user of the cartprovides answers through the input panel 76.

The system 10 may have certain components and software that are the sameor similar to a robotic system provided by the assignee InTouchTechnologies, Inc. of Goleta, Calif. under the name RP-7 and embodies asystem described in U.S. Pat. No. 6,925,357, which is herebyincorporated by reference.

FIG. 5 shows a display user interface (“DUI”) 120 that can be displayedat the remote station 14. The DUI 120 may include a robot view field 122that displays a video image captured by the robot camera and/or theoverhead camera. The DUI 120 may also include a station view field 124that displays a video image provided by the camera of the remote station14. The DUI 120 may be part of an application program stored andoperated by the computer 22 of the remote station 14.

The DUI 120 may include a graphical switch 126 that allows the user toselect between the video image provided by the robot camera and thevideo image provided by the overhead camera. The DUI 120 may also have agraphical switch 128 that allows the user to select the simultaneousdisplay of the video images from the robot and overhead cameras as shownin FIG. 6. The video images from both cameras can be streamed to theremote station from the cart. The images can be merged by presenting acenter rectangle of each image (e.g., 320×480 center area). A zoom orhighlighting feature may be utilized by manipulating a cursor on eitherimage. The system may also automatically pan a camera when the cursor ismoved out of the displayed field of view.

The system may automatically present the video image from a camera thathas an optimal view of an object. For example, the system may utilizepattern recognition techniques to determine which video image provides amore clear image of an object. The system may determine which camera isin closer proximity to an object and provide the image from the camerathat is closer to the object. The system may utilize positional feedbackfrom the cart to determine the proximity of the cameras relative to theobject. The system may also have sensors, such as laser, sonar, etc.that can determine the proximity of the cameras to the object. Thesystem may use the feedback and/or sensors to determine which camera iscloser to an object.

The system may automatically move the cameras so that each camera ispointed to the same or substantially the same field of view. Forexample, if the robot face is pointed toward an object, the overheadcamera can be automatically moved to capture a video image of the sameobject. Likewise, if the overhead camera is capturing a video image ofan object, the robot face can be automatically moved to point toward thesame object. This enhances the “presence” of the remote operator becausethey are facing the same object that the overhead camera is viewing.

The DUI 120 may have a graphical switch 130 that allows the user toswitch between sound captured by the robot microphone or the overheadmicrophone. The system may automatically switch between microphonesbased on a characteristic(s) of the sound captured by the microphones.For example, the system may switch to the microphone that provides thehighest aural clarity, or to the microphone that is in the closestproximity to a person or object generating the sound.

The DUI 120 may include a location display 138 that provides thelocation of the robot face. The CHANGE button 140 can be selected tochange the default robot face in a new session. The CHANGE button 140can be used to select and control a different robot face in a systemthat has multiple robot faces. The user can initiate and terminate asession by selecting box 142. The box 142 changes from CONNECT toDISCONNECT when the user selects the box to initiate a session. Systemsettings and support can be selected through buttons 144 and 146. Boththe robot view field 122 and the station view field 124 may haveassociated graphics to vary the video and audio displays. Each field mayhave an associated graphical audio slide bar 148 to vary the audio levelof a selected microphone and another slide bar 152 to vary the volume ofthe speakers.

The DUI 120 may have slide bars 150, 154 and 156 to vary the zoom, focusand brightness of a selected camera, respectively. A still picture maybe taken at either the robot face or remote station by selecting one ofthe graphical camera icons 158. The still picture may be the imagepresented at the corresponding field 122 or 124 at the time the cameraicon 158 is selected. Capturing and playing back video can be takenthrough graphical icons 160. A return to real time video can be resumed,after the taking of a still picture, captured video, or reviewing aslide show, by selecting a graphical LIVE button 162.

A still picture can be loaded from disk for viewing through selection oficon 164. Stored still images can be reviewed by selecting buttons 166.The number of the image displayed relative to the total number of imagesis shown by graphical boxes 168. The user can rapidly move through thestill images in a slide show fashion or move through a captured videoclip by moving the slide bar 170. A captured video image can be pausedthrough the selection of circle 174. Play can be resumed through thesame button 174. Video or still images may be dismissed from the activelist through button 172. Video or still images may be transferred to therobot by selecting icon 176. For example, a doctor at the remote stationmay transfer an x-ray to the screen of the robot.

The system may provide the ability to annotate 184 the image displayedin field 122 and/or 124. For example, a doctor at the remote station mayannotate some portion of the image captured by the robot face camera.The annotated image may be stored by the system. The system may alsoallow for annotation of images sent to the robot face through icon 176.For example, a doctor may send an x-ray to the robot face which isdisplayed by the robot screen. The doctor can annotate the x-ray topoint out a portion of the x-ray to personnel located at the robot site.This can assist in allowing the doctor to instruct personnel at therobot site.

The display user interface may include graphical inputs 186 that allowthe operator to turn the views of the remote station and remote camerason and off.

Referring to FIG. 1, the cart 12 can be used in an operating room. Byway of example, the boom 60 can be moved to place the overhead camera 18above an operating table 200. The overhead camera 18 may be locatedabove a sterile field. The robot face 16 may be placed adjacent to thesterile field. With such a configuration, personnel may conduct two-wayvideo conferencing through the robot face 16. The overhead camera 18 mayprovide a more desirable view of the patient and operating procedure.This would allow a physician at the remote station to view the procedureand have a video conference to provide instructions, mentoring, etc. topersonnel at the surgical site.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art.

What is claimed is:
 1. A tele-presence system, comprising: a cart, saidcart includes a robot face that has a robot monitor, a robot camera, arobot speaker, a robot microphone, and an overhead camera; and, a remotestation that is coupled to said robot face and said overhead camera,said remote station includes a station monitor, a station camera, astation speaker and a station microphone.
 2. The system of claim 1,wherein said remote station can display a video image from either saidrobot camera or said overhead camera.
 3. The system of claim 1, whereinsaid remote station monitor simultaneously displays a video image fromsaid robot camera and a video image from said overhead camera.
 4. Thesystem of claim 1, wherein said remote station determines an optimalview of an object and displays a video image from said robot camera orsaid overhead camera, whichever provides said optimal view.
 5. Thesystem of claim 1, wherein said robot camera automatically moves tocapture a video image of a field of view that is at least substantiallythe same as a field of view of said overhead camera.
 6. The system ofclaim 1, wherein said overhead camera automatically moves to capture avideo image of a field of view that is at least substantially the sameas a field of view of said robot camera.
 7. The system of claim 1,wherein said cart includes an overhead microphone.
 8. The system ofclaim 7, wherein said remote station includes an input that allows auser to switch between said robot microphone and said overheadmicrophone.
 9. The system of claim 7, wherein said sound generated bysaid station speakers is automatically switched between said robotmicrophone or said overhead microphone based on a characteristic ofsound captured by said robot and overhead microphones.
 10. The system ofclaim 1, wherein said cart includes an actuator that can vary a heightof said robot face.
 11. The system of claim 1, wherein said cartincludes an articulated arm that is coupled to and can move said robotface, and a boom that is coupled to said overhead camera.
 12. The systemof claim 11, wherein said robot face includes an input panel and can beoperated as a computer.
 13. The system of claim 11, wherein saidarticulated arm and said boom include position sensors.
 14. The systemof claim 11, wherein further comprising a laser pointer attached to saidboom.
 15. The system of claim 11, wherein said cart boom includes adetachable handle.
 16. A method for remotely viewing a field of view,comprising: moving a cart, the cart includes a robot face that has arobot monitor, a robot camera, a robot speaker, a robot microphone, andan overhead camera; and, transmitting to a remote station a video imagecaptured by the robot camera or a video image captured by the overheadcamera, the remote station includes a station monitor, a station camera,a station speaker and a station microphone.
 17. The method of claim 16,wherein the video images captured by the robot and overhead cameras areboth transmitted to the remote station.
 18. The method of claim 17,wherein the remote station monitor simultaneously displays the videoimage from the robot camera and the video image from the overheadcamera.
 19. The method of claim 16, wherein the remote stationdetermines an optimal view of an object and displays a video image fromrobot camera or the overhead camera, whichever provides the optimalview.
 20. The method of claim 16, further comprising automaticallymoving the robot camera to capture a video image of a field of view thatis at least substantially the same as a field of view of the overheadcamera.
 21. The method of claim 16, further comprising automaticallymoving the overhead camera to capture a video image of a field of viewthat is at least substantially the same as a field of view of the robotcamera.
 22. The method of claim 16, further comprising capturing soundwith an overhead microphone of the cart.
 23. The method of claim 22,further comprising selecting an input to switch between the robotmicrophone and the overhead microphone.
 24. The method of claim 22,further comprising automatically switching between the robot microphoneand the overhead microphone based on a characteristic of the soundcaptured by the robot and overhead microphones.
 25. The method of claim16, further comprising varying a height of the robot face.
 26. Themethod of claim 25, wherein the robot face is moved in two degrees offreedom.
 27. The method of claim 16, wherein the cart is moved within anoperating room.
 28. The method of claim 27, further comprising movingthe overhead camera above a sterile field.
 29. The method of claim 16,further comprising entering input into the robot face.
 30. The method ofclaim 16, further comprising replacing a detachable handle of the cart.